FusionProv: Towards a Provenance-Aware Distributed Filesystem

It has become increasingly important to capture and understand the origins and derivation of data (its provenance). A key issue in evaluating the feasibility of data provenance is its performance, overheads, and scalability. In this paper, we explore the feasibility of a management layer for parallel file systems, in which metadata includes both file operations and provenance metadata. We design and implement a provenance layer within a distributed file system —FusionFS, which implements a distributed file metadata management based on distributed hash tables. Our results show that FusionFS with its own storage layer for provenance capture is able to scale up to 1K nodes on BlueGene/P supercomputer.